Transformers and Attention

Before Transformers: The Sequence Problem

Text is fundamentally different from an image. An image has fixed spatial structure: pixels at position (x, y) are always there. A sentence is a sequence of variable length, and the relationship between words can span long distances. In "The trophy did not fit in the bag because it was too large," what does "it" refer to? The trophy, not the bag. Understanding this requires relating "it" to "trophy" across eight words of distance.

For years, the dominant approach to sequence data was Recurrent Neural Networks (RNNs). An RNN processes a sequence one element at a time, passing a "hidden state" from step to step. The hidden state at step 50 is supposed to remember what happened at step 1. The problem is it often does not. Information gets diluted as it passes through many sequential multiplications.

Long Short-Term Memory networks (LSTMs), introduced by Hochreiter and Schmidhuber in 1997, added gating mechanisms that helped retain important information over longer distances. They powered breakthrough results in machine translation and speech recognition through the 2010s. But LSTMs still had a fundamental bottleneck: the entire history of a sequence had to be compressed into a single fixed-size hidden state vector before generating each output token.

In 2015, Bahdanau, Cho, and Bengio introduced the first attention mechanism for sequence-to-sequence models, allowing the decoder to look directly at all encoder states rather than relying solely on the compressed hidden state. This was enormously effective for machine translation. It was also the seed of a more radical idea: what if the entire architecture was built around attention, with no recurrence at all?

The Core Idea: Attention

The intuition behind attention is something humans do naturally when reading. When you read the sentence "The scientist published the paper after she reviewed it carefully," and you want to understand what "she" refers to, your brain does not re-read the entire sentence. It pulls out the relevant word "scientist" and connects them. You are paying attention to specific parts of the input based on relevance to the current task.

In a Transformer, every token in a sequence is given the ability to directly attend to every other token, computing a weighted combination that reflects how relevant each position is. If "she" strongly attends to "scientist," the representation of "she" will be heavily influenced by the representation of "scientist," correctly capturing the coreference relationship.

The mechanism uses a query-key-value framework drawn from information retrieval. Think of it like a soft database lookup:

Q, K, and V are not separate inputs. They are all derived from the same token embeddings through three different learned linear projections. This means the model learns what to look for, what to advertise, and what to contribute, all from the same underlying representations.

Self-Attention Step by Step

Let us walk through the exact computation. Given a sequence of tokens, each represented as a vector of dimension d_model:

Step 1: Project into Q, K, V

Multiply each token embedding by three learned weight matrices (W_Q, W_K, W_V) to get three vectors per token: a query Q, a key K, and a value V. The dimensions of Q and K must match (call it d_k) since we will compute their dot product.

Step 2: Compute Attention Scores

For each token acting as a query, compute a dot product with every token's key. A high dot product means the query and key are aligned, that is, this pair of tokens has a strong relationship.

Step 3: Scale and Softmax

Divide each score by the square root of d_k. This scaling prevents the dot products from growing too large in magnitude (which would push the softmax into regions where gradients become tiny). Then apply softmax to get a probability distribution over all positions, the attention weights.

Step 4: Weighted Sum of Values

Multiply the attention weights by the value vectors and sum. The output for each token is a weighted mixture of all value vectors, where the weights reflect how relevant each position was. Tokens that scored highly in the attention contribute more to the output.

A Concrete Implementation

The mathematics of scaled dot-product attention can be implemented in about ten lines of Python:

import numpy as np

def softmax(x):
    # Subtract max for numerical stability — prevents overflow in exp()
    exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return exp_x / exp_x.sum(axis=-1, keepdims=True)

def scaled_dot_product_attention(Q, K, V):
    """
    Q: (seq_len, d_k) — queries
    K: (seq_len, d_k) — keys
    V: (seq_len, d_v) — values
    """
    d_k = Q.shape[-1]

    # Step 1: compute all pairwise scores — (seq_len, seq_len)
    scores = Q @ K.T / np.sqrt(d_k)

    # Step 2: apply softmax row-wise to get attention weights
    weights = softmax(scores)

    # Step 3: weighted sum of value vectors
    output = weights @ V

    return output, weights

# Example: 4 tokens, d_k = 4
np.random.seed(42)
seq_len, d_k = 4, 4

Q = np.random.randn(seq_len, d_k)
K = np.random.randn(seq_len, d_k)
V = np.random.randn(seq_len, d_k)

output, attn_weights = scaled_dot_product_attention(Q, K, V)

print("Attention weights (each row sums to 1):")
print(attn_weights.round(3))
print(f"\nOutput shape: {output.shape}")

This is the complete core of self-attention. Each row in the weight matrix is a distribution over all four positions. The output for each token is a weighted blend of all value vectors according to these weights.

Multi-Head Attention

A single attention head learns one way to relate tokens. But a sentence has multiple types of relationships simultaneously. "John gave Mary the book" contains relationships between giver and receiver, between action and object, between subject and verb. A single attention pattern cannot capture all of these at once.

Multi-head attention runs several attention operations in parallel, each with its own independent W_Q, W_K, and W_V projection matrices. Each "head" is free to learn a different type of relationship. The outputs from all heads are concatenated and projected through a final linear layer.

The original Transformer used 8 heads with d_model = 512. Each head operates in a reduced dimension of 512 / 8 = 64, so the total computational cost is comparable to a single head at full dimension. Research on analysing what individual heads learn has found that some heads specialise in syntactic relationships (subject-verb agreement), some in coreference (pronoun-noun), and some in positional patterns, though this specialisation is not guaranteed or required by the design.

The Transformer Block

Self-attention alone is not a complete model. The full Transformer block wraps it with several additional components, each serving a specific engineering purpose.

The 2017 Breakthrough: "Attention Is All You Need"

The Transformer architecture was introduced in the paper "Attention Is All You Need" by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, all researchers at Google at the time. It was presented at NeurIPS 2017.

The paper's central claim was bold: recurrence and convolution were not necessary for sequence modelling. Attention alone, applied in parallel across all positions, was sufficient and indeed superior. The model they proposed achieved state-of-the-art results on English-to-German and English-to-French machine translation tasks, while training far faster than the recurrent models of the time.

The base Transformer had these specs:

# Original Transformer "base" hyperparameters (Vaswani et al., 2017)
d_model    = 512   # embedding / model dimension
d_ff       = 2048  # feed-forward inner dimension (4× d_model)
num_heads  = 8     # attention heads per block
d_k = d_v  = 64   # dimension per head (512 / 8)
num_layers = 6     # number of encoder blocks (and 6 decoder blocks)
dropout    = 0.1   # dropout rate applied throughout
# Total parameters: approximately 65 million

The paper also introduced the concept of scaled dot-product attention. Removing the scaling factor 1/sqrt(d_k) causes the dot products to grow large in high dimensions, pushing softmax into saturation regions where gradients nearly vanish. The scaling was a practical and important engineering decision.

BERT and GPT: Two Strategies, One Architecture

The Transformer has an encoder stack and a decoder stack. Different research groups discovered that pre-training just one of these components on massive text datasets, then fine-tuning on specific tasks, produced remarkably capable models. Two paradigms emerged.

The Scale-Up Story

Using Transformers Today: Hugging Face

Training a Transformer from scratch requires enormous data and compute. In practice, you almost never do this. Instead, you use pre-trained models through the Hugging Face transformers library, the dominant open-source platform for working with Transformer models. It provides a unified API for hundreds of pre-trained models in dozens of languages.

from transformers import pipeline

# ── 1. Sentiment Analysis using a fine-tuned BERT model ───────────────
# Default model: distilbert-base-uncased-finetuned-sst-2-english
# (a compressed BERT fine-tuned on Stanford Sentiment Treebank)
classifier = pipeline("sentiment-analysis")

texts = [
    "I really enjoyed this course, the explanations are excellent!",
    "This was confusing and poorly structured.",
    "Convolutional neural networks are used in image recognition."
]

results = classifier(texts)
for text, result in zip(texts, results):
    print(f"Text: '{text[:50]}...'")
    print(f"  Label: {result['label']}, Score: {result['score']:.4f}\n")

# ── 2. Text Generation using GPT-2 ────────────────────────────────────
generator = pipeline("text-generation", model="gpt2")

prompt = "Artificial intelligence is transforming"
output = generator(
    prompt,
    max_new_tokens=50,       # generate 50 new tokens after the prompt
    num_return_sequences=1,
    do_sample=True,          # sample from the distribution (not greedy)
    temperature=0.8          # lower = more predictable, higher = more creative
)

print("Generated text:")
print(output[0]['generated_text'])

# ── 3. Question Answering using BERT ──────────────────────────────────
qa = pipeline("question-answering")

context = """
The Transformer architecture was introduced in 2017 by Vaswani et al.
in the paper 'Attention Is All You Need'. It relies entirely on
self-attention mechanisms and has become the foundation of modern
language models including BERT and GPT.
"""

result = qa(question="When was the Transformer introduced?", context=context)
print(f"\nQuestion: When was the Transformer introduced?")
print(f"Answer: {result['answer']} (confidence: {result['score']:.4f})")

What Just Happened?

The pipeline function downloaded a pre-trained model (several hundred MB), loaded its weights, and ran inference. For sentiment analysis, it used a DistilBERT model fine-tuned on human-labelled movie and review data. For question answering, it used a BERT model trained to extract spans from context paragraphs. None of the heavy lifting required any data or training from us.

Key Takeaways

• Recurrent networks process sequences step by step and struggle to maintain context over long distances. Transformers process all positions in parallel using attention.
• Self-attention lets every token attend to every other token in a sequence. The attention weight between two tokens reflects how relevant one is to the other.
• Scaled dot-product attention computes Q and K dot products, scales by 1/sqrt(d_k) to prevent large magnitudes, applies softmax to get weights, then takes a weighted sum of V vectors.
• Multi-head attention runs several attention heads in parallel. Each head can learn a different type of relationship (syntactic, semantic, positional). Outputs are concatenated and projected.
• A Transformer block wraps attention with a feed-forward network, residual connections, and Layer Normalisation. Blocks are stacked N times (6 in the original paper).
• Positional encoding adds sequence position information to token embeddings, since attention alone has no sense of word order.
• "Attention Is All You Need" (Vaswani et al., 2017, NeurIPS) introduced the architecture and showed recurrence was not needed for top-quality translation.
• BERT (encoder-only, bidirectional) excels at understanding tasks. GPT (decoder-only, causal) is designed for text generation. ChatGPT, GPT-4, and Claude are all GPT-style decoder models.
• The Hugging Face library lets you use pre-trained Transformers for dozens of NLP tasks with just a few lines of Python.